Adversarial approach for identifying inappropriate text content in images

ABSTRACT

An adversarial approach in detecting inappropriate text content in images. An expression from a listing of expressions may be selected. The listing of expressions may include words, phrases, or other textual content indicative of a particular type of message. Using the selected expression as a reference, the image is searched for a section that could be similar to the selected expression. The similarity between the selected expression and the section of the image may be in terms of shape. The section may be scored against the selected expression to determine how well the selected expression matches the section. The score may be used to determine whether or not the selected expression is present in the image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.60/872,928, filed on Dec. 4, 2006, which is incorporated herein byreference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to computer security, and moreparticularly but not exclusively to methods and apparatus foridentifying inappropriate text content in images.

2. Description of the Background Art

Electronic mail (“email”) has become a relatively common means ofcommunication among individuals with access to a computer network, suchas the Internet. Among its advantages, email is relatively convenient,fast, and cost-effective compared to traditional mail. It is thus nosurprise that a lot of businesses and home computer users have some formof email access. Unfortunately, the features that make email popularalso lead to its abuse. Specifically, unscrupulous advertisers, alsoknown as “spammers,” have resorted to mass emailings of advertisementsover the Internet. These mass emails, which are also referred to as“spam emails” or simply “spam,” are sent to computer users regardless ofwhether they asked for them or not. Spam includes any unsolicited email,not just advertisements. Spam is not only a nuisance, but also poses aneconomic burden.

Previously, the majority of spam consisted of text and images that arelinked to websites. More recently, spammers are sending spam with animage containing the inappropriate content (i.e., the unsolicitedmessage). The reason for embedding inappropriate content in an image isthat spam messages can be distinguished from normal or legitimatemessages in at least two ways. For one, the inappropriate content (e.g.,words such as “Viagra”, “free”, “online prescriptions,” etc.) can bereadily detected by keyword and statistical filters (e.g., see SahamiM., Dumais S., Heckerman D., and Horvitz E., “A Bayesian Approach toFiltering Junk E-mail,” AAAI'98 Workshop on Learning for TextCategorization, 27 Jul. 1998, Madison, Wis.). Second, the domain in URLs(uniform resource locators) in the spam can be compared to databases ofknown bad domains and links (e.g., see Internet URL www dot surbl dotorg).

In contrast, a spam email where the inappropriate content and URLs areembedded in an image may be harder to classify because the email itselfdoes not contain obvious spammy textual content and does not have alink/domain that can be looked up in a database of bad links/domains.

Using OCR (optical character recognition) techniques to identify spamimages (i.e., images having embedded spammy content) have been proposedbecause OCR can be used to identify text in images. In general, use ofOCR for anti-spam applications would involve performing OCR on an imageto extract text from the image, scoring the extracted text, andcomparing the score to a threshold to determine if the image containsspammy content. Examples of anti-spam applications that may incorporateOCR functionality include the SpamAssassin and Barracuda Networks spamfilters. Spammers responded to OCR solutions in spam filters with imagesdeliberately designed with anti-OCR features. Other approaches to combatspam images include flesh-tone analysis and use of regular expressions.

The present invention provides a novel and effective approach foridentifying content in an image even when the image has anti-OCRfeatures.

SUMMARY

In one embodiment, an expression from a listing of expressions isselected. The listing of expressions may include words, phrases, orother textual content indicative of a type of message. Using theselected expression as a reference, the image is searched for a sectionthat could be similar to the selected expression. The similarity betweenthe selected expression and the section of the image may be in terms ofshape. The section may be scored against the selected expression todetermine how well the selected expression matches the section. Thescore may be used to determine whether or not the selected expression ispresent in the image. Embodiments of the present invention may be usedin a variety of applications including antispam, anti-phishing,identification of confidential information in emails, identification ofcommunications that breach policies or regulations in emails, and othercomputer security applications involving identification of text contentin images.

These and other features of the present invention will be readilyapparent to persons of ordinary skill in the art upon reading theentirety of this disclosure, which includes the accompanying drawingsand claims.

DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example image included in a spam.

FIG. 2 shows text extracted from the image of FIG. 1 by opticalcharacter recognition.

FIG. 3 shows a schematic diagram of a computer in accordance with anembodiment of the present invention.

FIG. 4 shows a flow diagram of a method of identifying inappropriatetext content in images in accordance with an embodiment of the presentinvention.

FIG. 5 shows a flow diagram of a method of identifying inappropriatetext content in images in accordance with another embodiment of thepresent invention.

FIG. 6 shows a spam image included in an email and processed using themethod of FIG. 5.

FIG. 7 shows inappropriate text content found in the spam image of FIG.6 using the method of FIG. 5.

FIG. 8 shows a flow diagram of a method of identifying inappropriatetext content in images in accordance with yet another embodiment of thepresent invention.

The use of the same reference label in different drawings indicates thesame or like components.

DETAILED DESCRIPTION

In the present disclosure, numerous specific details are provided, suchas examples of apparatus, components, and methods, to provide a thoroughunderstanding of embodiments of the invention. Persons of ordinary skillin the art will recognize, however, that the invention can be practicedwithout one or more of the specific details. In other instances,well-known details are not shown or described to avoid obscuring aspectsof the invention.

FIG. 1 shows an example image included in a spam. The spam image of FIG.1 includes anti-OCR features in the form of an irregular background,fonts, and color scheme to confuse an OCR module. FIG. 2 shows the textextracted from the image of FIG. 1 using conventional OCR process. Theanti-OCR features fooled the OCR module enough to make the text largelyunintelligible, making it difficult to determine if the image containsinappropriate content, such as those commonly used in spam emails.

Referring now to FIG. 3, there is shown a schematic diagram of acomputer 300 in accordance with an embodiment of the present invention.The computer 300 may have less or more components to meet the needs of aparticular application. The computer 300 may include a processor 101,such as those from the Intel Corporation or Advanced Micro Devices, forexample. The computer 300 may have one or more buses 103 coupling itsvarious components. The computer 300 may include one or more user inputdevices 102 (e.g., keyboard, mouse), one or more data storage devices106 (e.g., hard drive, optical disk, USB memory), a display monitor 104(e.g., LCD, flat panel monitor, CRT), a computer network interface 105(e.g., network adapter, modem), and a main memory 108 (e.g., RAM). Inthe example of FIG. 1, the main memory 108 includes an antispam engine320, an OCR module 321, expressions 322, images 323, and emails 324. Thecomponents shown as being in the main memory 108 may be loaded from adata storage device 106 for execution or reading by the processor 101.For example, the emails 324 may be received over the Internet by way ofthe computer network interface 105, buffered in the data storage device106, and then loaded onto the main memory 108 for processing by theantispam engine 320. Similarly, the antispam engine 320 may be stored inthe data storage device 106 and then loaded onto the main memory 108 toprovide antispam functionalities in the computer 300.

The antispam engine 320 may comprise computer-readable program code foridentifying spam emails or other data with inappropriate content, whichmay comprise text that includes one or more words and phrases identifiedin the expressions 322. The antispam engine 320 may be configured toextract an image 323 from an email 324, use the OCR module 321 toextract text from the image 323, and process the extracted text outputto determine if the image 323 includes inappropriate content, such as anexpression 322. For example, the antispam engine 320 may be configuredto determine if one or more expressions in the expressions 322 arepresent in the extracted text. The antispam engine 320 may also beconfigured to directly process the image 323, without having to extracttext from the image 323, to determine whether or not the image 323includes inappropriate content. For example, the antispam engine 320 maydirectly compare the expressions 322 to sections of the image 323. Theantispam engine 320 may deem emails 324 with inappropriate content asspam.

The OCR module 321 may comprise computer-readable program code forextracting text from an image. The OCR module 321 may be configured toreceive an image in the form of an image file or other representationand process the image to generate text from the image. The OCR module321 may comprise a conventional OCR module. In one embodiment, the OCRmodule 321 is employed to extract embedded texts from the images 323,which in turn are extracted from the emails 324.

The expressions 322 may comprise words, phrases, terms, or othercharacter combinations or strings that may be present in spam images.Examples of such expressions may include “brokers,” “companyname”(particular companies), “currentprice,” “5daytarget,” “strongbuy,”“symbol,” “tradingalert” and so on. The expressions 322 may be obtainedfrom samples of confirmed spam emails, for example.

As will be more apparent below, embodiments of the present invention areadversarial in that they select an expression from the expressions 322and specifically look for the selected expression in the image, eitherdirectly or from the text output of the OCR module 321. That is, insteadof extracting text from an image and querying whether the extracted textis in a listing of expressions, embodiments of the present invention askthe question of whether a particular expression is in an image. Theadversarial approach allows for better accuracy in identifyinginappropriate content in images in that it focuses search for aparticular expression, allowing for more accurate reading of textembedded in images.

The emails 324 may comprise emails received over the computer networkinterface 105 or other means. The images 323 may comprise imagesextracted from the emails 324. The images 324 may be in any conventionalimage format including JPEG, TIFF, etc.

FIG. 4 shows a flow diagram of a method 400 of identifying inappropriatetext content in images in accordance with an embodiment of the presentinvention. FIG. 4 is explained using the components shown in FIG. 3.Other components may also be used without detracting from the merits ofthe present invention.

The method 400 starts after the antispam engine 320 extracts an image323 from an email 324. The antispam engine 320 then selects anexpression from the expressions 322 (step 401). Using the selectedexpression as a reference, the antispam engine 320 determines if thereis a section of the image 323 that corresponds to the start and end ofthe selected expression (step 402). That is, the selected expression isused as a basis in finding a corresponding section. For example, theantispam engine 320 may determine if the image 323 includes a sectionthat looks similar to the selected expression 322 in terms of shape. Theantispam engine 320 then compares the selected expression 322 to thesection to determine the closeness of the selected expression 322 to thesection. In one embodiment, this is performed by the antispam engine 320by scoring the section against the selected expression (step 403). Thescore may reflect how close the selected expression 322 is to thesection. For example, the higher the score, the higher the likelihoodthat the selected expression 322 matches the section. A minimumthreshold indicative of the amount of correspondence required to obtaina match between an expression 322 and a section may be predetermined.The value of the threshold may be obtained and optimized empirically. Ifthe score is higher than the threshold, the antispam engine 320 may deemthe selected expression 322 as being close enough to the section that amatch is obtained, i.e., the selected expression 322 is deemed found inthe image 323 (step 404). In that case, the antispam engine 320 recordsthat the selected expression was found at the location of the section inthe image 323. For each image 323, the antispam engine 320 may repeatthe above-described process for each of the expressions 322 (step 405).A separate scoring procedure may be performed for all identifiedexpressions 322 to determine whether or not the image is a spam image.For example, once the expressions 322 present in the image 323 have beenidentified, the antispam engine 320 may employ conventional text-basedalgorithms to determine if the identified expressions 322 are sufficientto deem the image 323 a spam image. The email 324 from which a spamimage was extracted may be deemed as spam.

FIG. 5 shows a flow diagram of a method 500 of identifying inappropriatetext content in images in accordance with another embodiment of thepresent invention. FIG. 5 is explained using the components shown inFIG. 3. Other components may also be used without detracting from themerits of the present invention.

The method 500 starts after the antispam engine 320 extracts an image323 from an email 324. The OCR module 321 then extracts text from theimage, hereinafter referred to as “OCR text output” (step 501). Theantispam engine 320 selects an expression from the expressions 322 (step502). Using the selected expression as a reference, the antispam engine320 finds an occurrence in the OCR text output that is suitably similarto the selected expression 322 (step 503). For example, the antispamengine 320 may find one or more occurrences in the OCR text output thatcould match the beginning and end of the selected expression 322 interms of shape. Conventional shape matching algorithms may be employedto perform the step 503. For example, the antispam engine may employ theshape matching algorithm disclosed in the publication “Shape Matchingand Object Recognition Using Shape Contexts”, S. Belongie, J. Malik, andJ. Puzicha., IEEE Transactions on PAMI, Vol 24, No. 24, April 2002.Other shape matching algorithms may also be employed without detractingfrom the merits of the present invention.

The antispam engine 320 determines the closeness of the selectedexpression 322 to each found occurrence, such as by assigning a scoreindicative of how well the selected expression 322 matches each foundoccurrence in the OCR text output (step 504). For example, the higherthe score, the higher the likelihood the selected expression 322 matchesthe found occurrence. The similarity between the selected expression 322and a found occurrence may be scored, for example, using the editdistance algorithm or the viterbi algorithm (e.g., see “UsingLexigraphical Distancing to Block Spam”, Jonathan Oliver, inPresentation of the Second MIT Spam Conference, Cambridge, Mass., 2005and “Spam deobfuscation using a hidden Markov model”, Honglak Lee andAndrew Y. Ng. in Proceedings of the Second Conference on Email andAnti-Spam (CEAS 2005)). Other scoring algorithms may also be usedwithout detracting from the merits of the present invention.

In the method 500, a minimum threshold indicative of the amount ofcorrespondence required to obtain a match between an expression 322 anda found occurrence may be predetermined. The value of the threshold maybe obtained and optimized empirically. If the score of the step 504 ishigher than the threshold, the antispam engine 320 may deem the selectedexpression 322 as being close enough to the occurrence that a match isobtained, i.e., the selected expression 322 is deemed found in the image323 (step 505). In that case, the antispam engine 320 records that theselected expression was found at the location of the occurrence in theimage 323. For each image 323, the antispam engine 320 may repeat theabove-described process for each of the expressions 322 (step 506). Aseparate scoring procedure may be performed for all identifiedexpressions 322 to determine whether or not the image is a spam image.For example, once the expressions 322 present in the image 323 have beenidentified, the antispam engine 320 may employ conventional text-basedalgorithms to determine if the identified expressions 322 are sufficientto deem the image 323 a spam image. The email 324 from which a spamimage was extracted may be deemed as spam.

FIG. 6 shows a spam image included in an email and processed using themethod 500. FIG. 7 shows the inappropriate text content found by themethod 500 on the spam image of FIG. 6. Note that the inappropriate textcontent, which is included in a list of expressions 322, has beensimplified for ease of processing by removing spaces between phrases.

FIG. 8 shows a flow diagram of a method 800 of identifying inappropriatetext content in images in accordance with yet another embodiment of thepresent invention. FIG. 8 is explained using the components shown inFIG. 3. Other components may also be used without detracting from themerits of the present invention.

The method 800 starts after the antispam engine 320 extracts an image323 from an email 324. The antispam engine 320 then selects anexpression from the expressions 322 (step 801). The antispam engine 320finds a section in the image 323 that is suitably similar to theselected expression 322 (step 802). For example, the antispam engine 320may find a section in the image 323 that could match the beginning andend of the selected expression 322 in terms of shape. A shape matchingalgorithm, such as that previously mentioned with reference to step 503of FIG. 5 or other conventional shape matching algorithm, may beemployed to perform the step 802.

The antispam engine 320 builds a text string directly (i.e., withoutfirst converting the image to text by OCR, for example) from the sectionof the image and then scores the text string against the selectedexpression to determine the closeness of the selected expression 322 tothe found section (step 803). The higher the resulting score, the higherthe likelihood the selected expression 322 matches the section. Forexample, to identify the text string, the antispam engine 320 mayprocess the section of the image 323 between the potential start and endpoints that could match the selected expression 322. The pixel blocks inbetween the potential start and end points (a region of connectedpixels) are then assigned probabilities of being the characters underconsideration (for example the characters in the ASCII character set).The pixel blocks in between the potential start and end points are thenscored using the aforementioned edit algorithm or viterbi algorithm todetermine the similarity of the selected expression 322 to the foundsection.

In the method 800, a minimum threshold indicative of the amount ofcorrespondence required to obtain a match between an expression 322 anda found section may be predetermined. The value of the threshold may beobtained and optimized empirically. If the score of the similaritybetween the selected expression 322 and the found section of the image323 is higher than the threshold, the antispam engine 320 may deem theselected expression 322 as being close enough to the found section thatthere is a match, i.e., the selected expression 322 is deemed found inthe image 323 (step 804). In that case, the antispam engine 320 recordsthat the selected expression was found at the location of the section inthe image 323. For each image 323, the antispam engine 320 may repeatthe above-described process for each of the expressions 322 (step 805).A separate scoring procedure may be performed for all identifiedexpressions 322 to determine whether or not an image is a spam image.For example, once the expressions 322 present in the image 323 have beenidentified, the antispam engine 320 may employ conventional text-basedalgorithms to determine if the identified expressions 322 are sufficientto deem the image 323 a spam image. The email 324 from which a spamimage was extracted may be deemed as spam.

In light of the present disclosure, those of ordinary skill in the artwill appreciate that embodiments of the present invention may beemployed in applications other than antispam. This is because theabove-disclosed techniques may be employed to identify text content inimages in general, the images being present in various types of messagesincluding emails, web page postings, electronic documents, and so on.For example, the components shown in FIG. 3 may be configured for otherapplications including anti-phishing, identification of confidentialinformation in emails, identification of communications that breachpolicies or regulations in emails, and other computer securityapplications involving identification of text content in images. Foranti-phishing applications, links to phishing sites may be included inthe expressions 322. In that case, the antispam engine 320 may beconfigured to determine if an image included in an email has textcontent matching a link to a phishing site included in the expressions322. Confidential (e.g., company trade secret information orintellectual property) or prohibited (e.g., text content that is againstpolicy or regulation) information may also be included in theexpressions 322 so that the antispam engine 320 may determine if suchinformation is present in an image included in an email message.

Improved techniques for identifying inappropriate text content in imageshave been disclosed. While specific embodiments of the present inventionhave been provided, it is to be understood that these embodiments arefor illustration purposes and not limiting. Many additional embodimentswill be apparent to persons of ordinary skill in the art reading thisdisclosure.

1. A method of identifying inappropriate text content in images, themethod to be performed using a computer and comprising: selecting anexpression from a listing of expressions; extracting an image from amessage; using the selected expression as a reference, finding a sectionof the image that corresponds to a start point and an end point of theselected expression; comparing the section of the image to the selectedexpression to determine how well the section of the image matches theselected expression; and determining if the selected expression ispresent in the section of the image based on the comparison of the imageto the selected expression; wherein comparing the section of the imageto the selected expression comprises assigning, to pixel blocks betweenthe start point and the end point of the section of the image,probabilities of being characters of the selected expression, andfurther comprising deeming the message as spam based on presence of theselected expression in the section of the image.
 2. The method of claim1 wherein finding the section of the image that corresponds to the startpoint and the end point of the selected expression comprises: finding asection of the image that has a shape similar to that of the selectedexpression.
 3. The method of claim 1 wherein comparing the section ofthe image against the selected expression comprises scoring the imageagainst the selected expression to generate a score.
 4. The method ofclaim 3 wherein determining if the selected expression is present in thesection of the image based on the comparison of the image to theselected expression comprises comparing the score to a threshold.
 5. Themethod of claim 4 wherein the selected expression is deemed present inthe section of the image when the score is higher than the threshold. 6.The method of claim 1 further comprising performing OCR on the image andwherein the section of the image comprises a text output of the OCR onthe image.
 7. The method of claim 1 further comprising building a textstring representing a text content of the section without performing OCRon the section.
 8. The method of claim 7 wherein comparing the sectionof the image to the selected expression comprises comparing the textstring to the selected expression.
 9. The method of claim 1 wherein themessage comprises an email and the selected expression comprises a wordor phrase indicative of spam.